{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "cDgpEHaUETGA"
      },
      "source": [
        "##### Copyright 2019 Google LLC.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "esgwBPvbEVsD"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YMDobAW8mfWo"
      },
      "source": [
        "# Non-rigid surface deformation\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/graphics/blob/master/tensorflow_graphics/notebooks/non_rigid_deformation.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/graphics/blob/master/tensorflow_graphics/notebooks/non_rigid_deformation.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "L9EFL7fhiHbD"
      },
      "source": [
        "Non-rigid surface deformation is a technique that, among other things, can be used to interactively manipulate meshes or to deform a template mesh to fit to a point-cloud. When manipulating meshes, this can for instance allow users to move the hand of a character, and have the rest of the arm deform in a realistic manner. It is interesting to note that the deformation can also be performed over the scale of parts or the entire mesh.\n",
        "\n",
        "![](https://storage.googleapis.com/tensorflow-graphics/notebooks/non_rigid_deformation/task.jpg)\n",
        "\n",
        "This notebook illustrates how to use [Tensorflow Graphics](https://github.com/tensorflow/graphics) to perform deformations similiar to the one contained in the above image. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BSMKlF6nAEPE"
      },
      "source": [
        "## Setup \u0026 Imports\n",
        "\n",
        "If Tensorflow Graphics is not installed on your system, the following cell can install the Tensorflow Graphics package for you."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "gpmGP3DgAKRe"
      },
      "outputs": [],
      "source": [
        "!pip install tensorflow_graphics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-YxmozRVANeD"
      },
      "source": [
        "Now that Tensorflow Graphics is installed, let's import everything needed to run the demo contained in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "-OAKdOoTf_-f"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow_graphics.geometry.deformation_energy import as_conformal_as_possible\n",
        "from tensorflow_graphics.geometry.representation.mesh import utils as mesh_utils\n",
        "from tensorflow_graphics.geometry.transformation import quaternion\n",
        "from tensorflow_graphics.math.optimizer import levenberg_marquardt\n",
        "from tensorflow_graphics.notebooks import threejs_visualization\n",
        "from tensorflow_graphics.notebooks.resources import triangulated_stripe"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1jLD48VvMquE"
      },
      "source": [
        "In this example, we build a mesh that corresponds to a flat and rectangular surface. Using the sliders, you can control the position of the deformation constraints applied to that surface, which respectively correspond to all the points along the left boundary, center, and right boundary of the mesh."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "IpztINWQ4Q4H"
      },
      "outputs": [],
      "source": [
        "mesh_rest_pose = triangulated_stripe.mesh\n",
        "connectivity = mesh_utils.extract_unique_edges_from_triangular_mesh(triangulated_stripe.mesh['faces'])\n",
        "camera = threejs_visualization.build_perspective_camera(\n",
        "    field_of_view=40.0, position=(0.0, -5.0, 5.0))\n",
        "width = 500\n",
        "height = 500\n",
        "_ = threejs_visualization.triangular_mesh_renderer([mesh_rest_pose],\n",
        "                                                   width=width,\n",
        "                                                   height=height,\n",
        "                                                   camera=camera)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "JVSAgVRXypak"
      },
      "outputs": [],
      "source": [
        "###############\n",
        "# UI controls #\n",
        "###############\n",
        "#@title Constraints on the deformed pose { vertical-output: false, run: \"auto\" }\n",
        "constraint_1_z = 0  #@param { type: \"slider\", min: -1, max: 1 , step: 0.05 }\n",
        "constraint_2_z = -1  #@param { type: \"slider\", min: -1, max: 1 , step: 0.05 }\n",
        "constraint_3_z = 0  #@param { type: \"slider\", min: -1, max: 1 , step: 0.05 }\n",
        "\n",
        "vertices_rest_pose = tf.Variable(mesh_rest_pose['vertices'])\n",
        "vertices_deformed_pose = np.copy(mesh_rest_pose['vertices'])\n",
        "num_vertices = vertices_deformed_pose.shape[0]\n",
        "\n",
        "# Adds the user-defined constraints\n",
        "vertices_deformed_pose[0, 2] = constraint_1_z\n",
        "vertices_deformed_pose[num_vertices // 2, 2] = constraint_1_z\n",
        "vertices_deformed_pose[num_vertices // 4, 2] = constraint_2_z\n",
        "vertices_deformed_pose[num_vertices // 2 + num_vertices // 4, 2] = constraint_2_z\n",
        "vertices_deformed_pose[num_vertices // 2 - 1, 2] = constraint_3_z\n",
        "vertices_deformed_pose[-1, 2] = constraint_3_z\n",
        "\n",
        "mesh_deformed_pose = {\n",
        "    'vertices': vertices_deformed_pose,\n",
        "    'faces': mesh_rest_pose['faces']\n",
        "}\n",
        "\n",
        "vertices_deformed_pose = tf.Variable(vertices_deformed_pose)\n",
        "\n",
        "# Builds a camera and render the mesh.\n",
        "camera = threejs_visualization.build_perspective_camera(\n",
        "    field_of_view=40.0, position=(0.0, -5.0, 5.0))\n",
        "_ = threejs_visualization.triangular_mesh_renderer([mesh_rest_pose],\n",
        "                                                   width=width,\n",
        "                                                   height=height,\n",
        "                                                   camera=camera)\n",
        "_ = threejs_visualization.triangular_mesh_renderer([mesh_deformed_pose],\n",
        "                                                   width=width,\n",
        "                                                   height=height,\n",
        "                                                   camera=camera)\n",
        "\n",
        "geometries = threejs_visualization.triangular_mesh_renderer(\n",
        "    [mesh_deformed_pose], width=width, height=height, camera=camera)\n",
        "\n",
        "\n",
        "################\n",
        "# Optimization #\n",
        "################\n",
        "def update_viewer_callback(iteration, objective_value, variables):\n",
        "  \"\"\"Callback to be called at each step of the optimization.\"\"\"\n",
        "  geometries[0].getAttribute('position').copyArray(\n",
        "      variables[0].numpy().ravel().tolist())\n",
        "  geometries[0].getAttribute('position').needsUpdate = True\n",
        "  geometries[0].computeVertexNormals()\n",
        "\n",
        "\n",
        "def deformation_energy(vertices_deformed_pose, rotation):\n",
        "  \"\"\"As conformal as possible deformation energy.\"\"\"\n",
        "  return as_conformal_as_possible.energy(\n",
        "      vertices_rest_pose,\n",
        "      vertices_deformed_pose,\n",
        "      rotation,\n",
        "      connectivity,\n",
        "      aggregate_loss=False)\n",
        "\n",
        "\n",
        "def soft_constraints(vertices_deformed_pose):\n",
        "  \"\"\"Soft constrains forcing results to obey the user-defined constraints.\"\"\"\n",
        "  weight = 10.0\n",
        "  return (\n",
        "      weight * (vertices_deformed_pose[0, 2] - constraint_1_z),\n",
        "      weight * (vertices_deformed_pose[num_vertices // 2, 2] - constraint_1_z),\n",
        "      weight * (vertices_deformed_pose[num_vertices // 4, 2] - constraint_2_z),\n",
        "      weight * (vertices_deformed_pose[num_vertices // 2 + num_vertices // 4, 2] -\n",
        "                constraint_2_z),\n",
        "      weight *\n",
        "      (vertices_deformed_pose[num_vertices // 2 - 1, 2] - constraint_3_z),\n",
        "      weight * (vertices_deformed_pose[-1, 2] - constraint_3_z),\n",
        "  )\n",
        "\n",
        "\n",
        "def fitting_energy(vertices_deformed_pose, rotation):\n",
        "  deformation = deformation_energy(vertices_deformed_pose, rotation)\n",
        "  constraints = soft_constraints(vertices_deformed_pose)\n",
        "  return tf.concat((deformation, constraints), axis=0)\n",
        "\n",
        "\n",
        "rotations = tf.Variable(quaternion.from_euler(np.zeros((num_vertices, 3))))\n",
        "\n",
        "max_iterations = 15  #@param { isTemplate: true, type: \"integer\" }\n",
        "_ = levenberg_marquardt.minimize(\n",
        "    residuals=fitting_energy,\n",
        "    variables=(vertices_deformed_pose, rotations),\n",
        "    max_iterations=int(max_iterations),\n",
        "    callback=update_viewer_callback)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "non_rigid_deformation.ipynb",
      "provenance": [],
      "toc_visible": true,
      "private_outputs": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
